Landing site localization for dynamic control of an aircraft toward a landing site

ABSTRACT

A system having components coupled to an aircraft and components remote from the aircraft processes sensor-derived data, transmits information between aircraft system components and remote system components, and dynamically generates updated analyses of position and orientation of the aircraft relative to a desired landing site, while the aircraft is in flight toward the desired landing site. Based on the position and orientation information, the system generates instructions for flight control of the aircraft toward a flight path to the landing site, and can update flight control instructions as new data is received and processed.

BACKGROUND

This disclosure relates generally to landing site localization, and morespecifically to determining the location and orientation of an aircraftrelative to a landing site (e.g., airport runway) during flight of theaircraft.

The final phases of landing an aircraft are critical, and successfullanding requires that the distance to, lateral offset from, elevationoffset from, and orientation of an aircraft relative to the landing siteto be known to a high degree of certainty. Current systems fornavigation (e.g., navigation to an airport, navigation in the vicinityof an airport) require installation and maintenance of expensiveapparatus, lack the precision required for automated landing procedures,are not reliable to a high enough degree, and/or are prone tointerference. The inventions described herein relate to improved systemsand methods for landing site localization, and can be used for executingan automated aircraft landing at a desired landing site.

SUMMARY

While an aircraft is in flight toward the desired landing site, a systemhaving components coupled to the aircraft and components remote from theaircraft processes sensor-derived data, transmits information betweenaircraft system components and remote system components, and dynamicallygenerates updated analyses of position and orientation of the aircraftrelative to a desired landing site. Based on the position andorientation information, the system generates instructions for flightcontrol of the aircraft toward a flight path to the landing site, andcan update flight control instructions as new data is received andprocessed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic of a system for landing site localization, inaccordance with one or more embodiments.

FIG. 1B is a schematic of a variation of a system for landing sitelocalization, in accordance with one or more embodiments.

FIG. 2A depicts a flowchart of a method 200 for landing sitelocalization, in accordance with one or more embodiments.

FIG. 2B depicts a schematics of a method flow according to embodimentsshown in FIG. 2A.

FIG. 2C depicts a flowchart of a portion of a method for landing sitelocalization, in accordance with one or more embodiments.

FIG. 3A depicts a schematic of a variation of a portion of the methodshown in FIGS. 2A-2B.

FIG. 3B depicts a schematic of another variation of a portion of themethod shown in FIGS. 2A-2B.

FIG. 3C depicts a schematic of an alternative variation to the variationof the portion of the method shown in FIG. 3B.

FIG. 3D depicts a schematic of another variation of a portion of themethod shown in FIGS. 2A-2B.

FIG. 4A depicts an embodiment of a method for camera subsystemcalibration, in accordance with one or more embodiments.

FIG. 4B depicts a schematic of an embodiment of the method shown in FIG.4A.

FIG. 5 depicts a method for performing a system check, in accordancewith one or more embodiments.

The figures depict various embodiments for purposes of illustrationonly. One skilled in the art will readily recognize from the followingdiscussion that alternative embodiments of the structures and methodsillustrated herein may be employed without departing from the principlesdescribed herein.

DETAILED DESCRIPTION 1. System for Landing Site Localization

FIG. 1A depicts a schematic of a system 100 for landing sitelocalization, in accordance with one or more embodiments. The system 100includes one or more flight data subsystems 110 coupled to (e.g.,mounted to, onboard, within, etc.) an aircraft 105, a remote station 120in communication with a data center 130 at a location remote from theaircraft 105, and an operator interface 140 in communication with theremote station 120 by way of the data center 130. The system 100 canalso include a flight management system (FMS) 150 including interfacesbetween the remote station 120 to the FMS 150 and/or interfaces betweenthe flight data subsystems 110 and the FMS 150. The system 100 providesstructures, subsystem interfaces, and operation modes useful forimplementation of automated flight operations, including operationsassociated with methods described in more detail in Section 2 below.

1.1 System—Aircraft

The aircraft 105 shown in FIG. 1 is a fixed-wing aircraft. The aircrafthas flight control surfaces for aerodynamically affecting flight of theaircraft relative to a pitch axis (i.e., a transverse axis), a yaw axis(i.e., a vertical axis), and a roll axis (i.e., longitudinal axis) ofthe aircraft. Flight control surfaces can include one or more of:ailerons, flaps, elevators, stabilizers (e.g., horizontal stabilizers),rudders, spoilers, slats, air brakes, vortex generators, trim surfaces,and any other suitable control surfaces. The aircraft also has a powerplant for generation of mechanical power associated with flightoperations, and in variations, the power plant can include one or moreof: a piston engine (e.g., in-line engine, V-type engine, opposedengine, radial engine, etc.), a gas turbine engine (e.g., turbojetengine, turbofan engine), a pulse jet, a rocket, a Wankel engine, aDiesel engine, an electric engine, a hybrid engine, and any othersuitable power plant system. The power plant is coupled to an energysource (e.g., fuel system, battery, solar cell, etc.) and a coolingsystem (e.g., forced convection cooling system, liquid cooling system,oil cooling system, etc.) for aircraft performance in flight.

While this description uses a fixed-wing aircraft as an example, theprinciples described herein are equally applicable to variations of theaircraft 105 including form factors and/or control surfaces associatedwith one or more of: rotorcraft, gliders, lighter-than-air aircraft(e.g., airships, balloons), powered-lift aircraft, powered-parachuteaircraft, weight-shift-control aircraft, rockets, and/or any othersuitable types of aircraft. Still other variations of the system 100 caninvolve terrestrial vehicles, water vehicles, amphibious vehicles, orother non-aircraft vehicles.

1.2 System—Flight Data Subsystems

The flight data subsystems 110 include subsystems capable of generatingdata associated with dynamic states of the aircraft, environments aboutthe aircraft, operation states of aircraft systems (e.g., power plantsystems, energy systems, electrical systems, etc.), and any othersuitable systems associated with operations of the aircraft on theground or in flight. The flight data subsystems 110 also includesubsystems capable of transmitting data to and from the aircraft 105 andother remote systems.

As such, the flight data subsystems 110 include subsystems that generateand receive information generated from subsystems coupled to theaircraft 105, as well as a flight computer 115 providing computationalinfrastructure (e.g., processing components, communication buses,memory, etc.) for communicating data between the subsystems. The flightcomputer 115 thus provides architecture for communication of datagenerated by subsystems, for communication with other systems remotefrom the aircraft 105, for control of subsystems, and/or for control ofthe aircraft. The flight data subsystems 110 can thus includespecialized computer components designed for use in an aircraft, and inparticular, can include components that are customized in configurationrelative to each other and customized in relation to processing ofsignals received and processed to perform aspects of the methodsdescribed in Section 2 below.

Information routed between the flight data subsystems 110 and othersystems remote from the aircraft 105 can optionally be routed through aflight management system (FMS) 150, configured for automation of flighttasks in relation to a flight plan. The FMS 150 processes navigationdatabase information (e.g., information associated with waypoints,airways, navigation aids, airports, runways, departure procedures,arrival procedures, holding patterns, etc.), aircraft subsystemstatuses, and outputs of other subsystems (e.g., radar subsystems,sensor subsystems) and determines one or more desired flight paths basedon the information. The FMS can cooperate with the flight computer 115in receiving outputs of other subsystems of the flight data subsystems110 and/or transmitting control instructions to affect operationalstates of other subsystems of the flight data subsystems 110. The FMS150 can also include or interface with other control systems (e.g., ofan autopilot) to transform calculated flight information intoinstructions for control of control surfaces of the aircraft 105including one or more of: ailerons, flaps, elevators, stabilizers (e.g.,horizontal stabilizers), rudders, spoilers, slats, air brakes, vortexgenerators, trim surfaces, and any other suitable control surfaces.

1.2.1 System—Flight Data Subsystems—Camera and IMU Components

As shown in FIG. 1A, the flight data subsystems 110 include a camerasubsystem 111 mounted to the aircraft, where the camera subsystem 111includes sensors configured to capture features of the landing site,features of objects in the vicinity of the landing site, features ofcalibration objects along a path of operation of the aircraft, featuresof other objects along a path of operation of the aircraft, and/or anyother suitable object aspects to facilitate automated landing of theaircraft at a desired landing site.

Sensors of the camera subsystem 111 can utilize the visible spectrum.Sensors of the camera subsystem 111 can additionally or alternativelyinclude longwave infrared (LWIR) sensors (e.g., sensors operating in the8-12 μm band). The camera subsystem 111 can also include opticalelements (e.g., lenses, filters, mirrors, apertures etc.) formanipulating light reaching the sensors of the camera subsystem 111. Inrelation to detection of airport lighting systems for landing sitelocalization relative to airport lighting, the camera subsystem 111 caninclude one or more filters optically coupled to the sensors andconfigured to detect spectra of light emitted from airfield landingsystems (e.g., lighting systems in accordance with Federal AviationAdministration Advisory Circular 150/5345-46E). Variations of the camerasubsystem 111 can, however, have any other suitable sensor types and/oroptical elements associated with visible spectra and/or non-visiblespectra electromagnetic radiation.

The camera subsystem 111 can have one or more cameras structurallymounted to the aircraft and positioned so as to enable detection of thelanding site or other site relevant to operation of the aircraft, as theaircraft traverses through space. Multiple cameras can be used forsystem redundancy (e.g., in the event a subset of cameras have occludedoptical elements) and/or for providing different field of view optionsdepending on approach path and orientation to a landing site. Thecamera(s) of the camera subsystem 111 can be coupled to an interiorportion of the aircraft 105, or can be coupled to an exterior portion ofthe aircraft 105. Mounting positions are associated with desired flightpaths to a landing site (e.g., approach patterns, instructions from airtraffic control, etc.). As such, the camera subsystem 111 can have acamera that has a field of view of at least 270 degrees about theaircraft 105. The camera subsystem 111 can additionally or alternativelyhave a first camera mounted toward a port side of the aircraft (e.g.,for left traffic operations), a second camera mounted toward a starboardside of the aircraft (e.g., for right traffic operations), a thirdcamera mounted toward a nose portion of the aircraft (e.g., forstraight-in approaches), and/or any other suitable cameras mounted atany other suitable portion of the aircraft 105.

The camera(s) of the camera subsystem 111 can thus be fixed in position.The camera(s) of the camera subsystem 111 can alternatively beadjustable in position based on flight paths of the aircraft 105 to thelanding site. The camera subsystem 111 can thus include actuatorscoupled to the camera(s) of the camera subsystem 111 and/or positionencoders coupled to the actuators, in relation to electronic control ofcamera positions. In relation to image stabilization, the camera(s) ofthe camera subsystem 111 can be coupled to image stabilizationsubsystems (e.g., gimbals) to reduce artifacts due to vibration or otherundesired image artifacts that would otherwise be included in image datagenerated from the camera subsystems 111.

The camera subsystem 111 produces output images that have acharacteristic resolution (e.g., associated with a sensor size), focallength, aspect ratio, and/or directionality (e.g., unidirectionalityassociated with 360 degree images), format color model, depth, and/orother aspects. The camera subsystem 111 can be configured for one ormore of: monoscopic images, stereoscopic images, panoramic images,and/or any other suitable type of image output. Furthermore, whileimages are described, the camera subsystem 111 can be configured tooutput video data, in relation to the method(s) described in Section 2below.

In one variation, as shown in FIG. 1B, the camera subsystem 111 includesa first camera 111 a mounted at a port side of the aircraft 105 and asecond camera 111 b mounted at a starboard side of the aircraft, wherethe first camera 111 a and the second camera 111 b collectively havemodes for generation of stereoscopic images (e.g., left and rightstereoscopic images associated with similar or identical time points ofimage capture). Stereoscopic images can then be transmitted to anoperator wearing a head mounted display (HMD) 140 b or otherwiseinteracting with a display of an operator interface for viewingstereoscopic images (e.g., in a 3D image format).

The flight data subsystem 110 also includes one or more inertialmeasurement units (IMUs) 112 for measuring and outputting dataassociated with the aircraft's specific force, angular rate, magneticfield surrounding the aircraft 105, and/or other position, velocity, andacceleration-associated data. Outputs of the IMU can be processed withoutputs of other aircraft subsystem outputs to determine poses of theaircraft 105 relative to a landing site (or other target), and/or posetrajectories of the aircraft 105 relative to a landing site (or othertarget). The IMU 112 includes one or more accelerometers, one or moregyroscopes, and can include one or more magnetometers, where any or allof the accelerometer(s), gyroscope(s), and magnetometer(s) can beassociated with a pitch axis, a yaw axis, and a roll axis of theaircraft 105.

The IMUS 112 are coupled to the aircraft, and can be positioned internalto the aircraft or mounted to an exterior portion of the aircraft. Inrelation to measurement facilitation and/or post-processing of data formthe IMU, the IMU can be coupled to a vibration dampener for mitigationof data artifacts from sources of vibration (e.g., engine vibration) orother undesired signal components.

Additionally or alternatively, the system 100 can include a radarsubsystem that operates to detect radar responsive (e.g., reflective,scattering, absorbing, etc.) objects positioned relative to a flightpath of the aircraft 105 (e.g., below the aircraft 105), in order tofacilitate determination of pose or state of the aircraft 105 insupplementing methods described below. Additionally or alternatively,the system can include a light emitting subsystem that operates todetect light responsive (e.g., reflective, scattering, absorbing, etc.)objects positioned relative to a flight path of the aircraft 105 (e.g.,below the aircraft 105), in order to facilitate determination of pose orstate of the aircraft 105 in supplementing methods described below.

1.2.2 System—Flight Data Subsystems—Communication Components

The flight data subsystem 110 also includes a radio transmissionsubsystem 113 for communication with the aircraft 105, for transmissionof aircraft identification information, or for transmission of othersignals. The radio transmission subsystem 113 can include one or moremultidirectional radios (e.g., bi-directional radios) onboard theaircraft, with antennas mounted to the aircraft in a manner that reducessignal transmission interference (e.g., through other structures of theaircraft). The radios of the radio transmission subsystem 113 operate inapproved frequency bands (e.g., bands approved through FederalCommunications Commission regulations, bands approved through FederalAviation Administration advisory circulars, etc.).

The flight data subsystem 110 can also include a satellite transmissionsubsystem 114 for interfacing with one or more satellites includingsatellite 14. The satellite transmission subsystem 114 transmits and/orreceives satellite data for navigation purposes (e.g., on a scaleassociated with less precision than that used for landing at a landingsite), for traffic avoidance in coordination with automatic dependentsurveillance broadcast (ADS-B) functionality, for weather services(e.g., in relation to weather along flight path, in relation to windsaloft, in relation to wind on the ground, etc.), for flight information(e.g., associated with flight restrictions, for notices, etc.), and/orfor any other suitable purpose. The satellite transmission subsystem 114operates in approved frequency bands (e.g., bands approved throughFederal Communications Commission regulations, bands approved throughFederal Aviation Administration advisory circulars, etc.).

The communication-related components of the flight data subsystems 110can additionally or alternatively cooperate with or supplement data fromother avionics components (e.g., a global positioning system and/orother localization subsystem 116), electrical components (e.g., lights),and/or sensors that support flight operations (e.g., in flight, duringlanding, on the ground, etc.), that support observability by othertraffic, that support observability by other aircraft detection systems,that provide environmental information (e.g., pressure information,moisture information, visibility information, etc.) and/or perform otherfunctions related to aircraft communications and observability.

1.3 System—Remote Components

As shown in FIG. 1A, the system 100 also includes a remote station 120that includes devices for wirelessly receiving data from andtransmitting data to subsystems coupled to the aircraft. The remotestation 120 includes one or more multidirectional radios (e.g.,bi-directional radios) onboard the aircraft, with antennas mounted tothe aircraft in a manner that reduces signal transmission interference(e.g., through other structures of the aircraft). The radios of theremote station operate in approved frequency bands (e.g., bands approvedthrough Federal Communications Commission regulations, bands approvedthrough Federal Aviation Administration advisory circulars, etc.). Theremote station 120 is in communication with a data center 130 forstorage and retrieval of data derived from subsystems of the aircraft105 and/or outputs from the operator interface 140 described in moredetail below. The data center uses storage and retrieval protocols andcan use data encryption protocols for promoting security in relation tohandling sensitive information pertaining to autonomous flight of theaircraft 105.

The remote station 120 can also use communications technologies and/orprotocols in relation to data transmission operations with the datacenter 130, subsystems of the aircraft 105, and/or the operatorinterface 140 described in more detail below. For example, the remotestation 120 can have communication links using technologies such asEthernet, 802.11, worldwide interoperability for microwave access(WiMAX), 3G, 4G, code division multiple access (CDMA), digitalsubscriber line (DSL), or other communication technologies. Examples ofnetworking protocols used for communications with the remote station 120include user datagram protocol (UDP) and/or any other suitable protocol.Data exchanged with the remote station 120 can be represented using anysuitable format.

Furthermore, in relation to communications-related subsystems, if acommunications do not operate as intended (e.g., a communication linkfails), the aircraft 105 can be transitioned into a safety operationmode. In an example, in the safety operation mode, the aircraft 105enters a holding pattern until operation of the communications-relatedsubsystems are restored to proper operation, or until the aircraft 105can be operated safely/safely landed in another manner.

As shown in FIG. 1A, the system 100 also includes an operator interface140. The operator interface 140 receives processed data (e.g., imagedata) generated from the subsystems of the aircraft 105, providesrepresentations of processed data to an operator or other entity (e.g.,through a display), and receives inputs provided by the operator orother entity in response to provided representations of processed data.The operator interface 140 can include a conventional computer system,such as a desktop or laptop computer. Additionally or alternatively, theoperator interface 140 can include a device having computerfunctionality, such as a personal digital assistant (PDA), a mobiletelephone, a smartphone, a wearable computing device (e.g., awrist-borne wearable computing device, a head-mounted wearable computingdevice, etc.), or another suitable device. The operator interface 140 iselectronically coupled to the remote station 120 and/or the data center130 by any combination of local area and/or wide area networks, usingtransmission and storage protocols, as described above, and can use bothwired and/or wireless communication systems.

The operator interface 140 can include a display for presentation ofvisually-observed digital content (e.g., images/videos from camerasubsystem components of the aircraft 105), as shown in FIG. 1A. Theoperator interface can additionally or alternatively include a headmounted display 140 b as shown in FIG. 1B for presentation of content tothe operator, as described above. In relation to input devices, theoperator interface 140 can include one or more of: a touch pad, a touchscreen, a mouse, a joystick, an audio input device, an optical inputdevice, and any other suitable input device for receiving inputs fromthe user.

Portions of one or more of: the flight computer 115 onboard the aircraft105, the FMS 150, the remote station 120, the data center 130, and theoperator interface 140 can operate as a computing system that includesmachine-readable instructions in non-transitory media for implementationof an embodiment of the method 200 described below, in relation to oneor more of: transmitting an image taken from the camera subsystem 111and capturing a landing site; receiving a reference position of areference object associated with the landing site within the image; fromthe reference position, generating an image-estimated pose of theaircraft 105; updating a pose trajectory of the aircraft 105 uponprocessing the image-estimated pose and an output from the IMU 112having a time stamp corresponding to the image; and based upon the posetrajectory, generating a set of instructions for flight control of theaircraft toward a flight path to the landing site (e.g., with the FMS150, etc.). In relation to flight control, the system 100 can include anelectronic interface between the remote station 110 and a flightmanagement system of the aircraft (e.g., as supported by the computingsystem), the electronic interface operable in a mode that transmits theset of instructions to the flight management system and controls flightof the aircraft toward the flight path. Additional aspects of the method200 are described in further detail in Section 2 below.

Further, while the system(s) described above can implement embodiments,variations, and/or examples of the method(s) described below, thesystem(s) can additionally or alternatively implement any other suitablemethod(s).

2. Method for Landing Site Localization

FIG. 2A depicts a flowchart of a method 200 for landing sitelocalization, in accordance with one or more embodiments. FIG. 2Bdepicts a schematics of a method flow according to embodiments shown inFIG. 2A. The method 200 functions to process sensor-derived data,transmit information between aircraft subsystems and systems remote fromthe aircraft, and dynamically generate updated estimates of position andorientation of the aircraft relative to a desired landing site, whilethe aircraft is in flight toward the desired landing site. Based on theposition and orientation information, the method 200 can also generateinstructions for flight control of the aircraft toward a flight path tothe landing site, and can update flight control instructions as new datais received and processed. The method 200 can also include functionalityfor directly controlling flight of the aircraft toward the landing sitein a reliable and safe manner. The method 200 can be implemented by oneor more embodiments of the system 100 described above, in relation toFIGS. 1A and 1B. In particular, portions of the method 200 can beimplemented by the computing system components described above, forinstance, at a portion of the computing system operating at the remotestation and/or at a portion of the computing system operating at aflight computer onboard the aircraft, with communication of inputs andoutputs across computing system components as defined by thearchitecture described above.

2.1 Method—Receiving Data from Aircraft

As shown in FIGS. 2A and 2B, Blocks 210 a and 210 b includefunctionality for receiving an image taken from a camera subsystemcoupled to an aircraft, where the image captures a landing site within afield of view of the camera subsystem. In particular, in relation tosystem elements described above, in 210 a and 210 b the remote stationwirelessly receives one or more images taken from cameras of the camerasubsystem coupled to the aircraft during flight of the aircraft in thevicinity of the landing site. Transmission of the images can occurthrough data transmission systems of the aircraft and remote station.The remote station and/or other computing system can then process thereceived image(s) according to subsequent blocks of the method.

In relation to image type, sensors of the camera subsystem involved inimage capture can generate visible spectrum images and/or non-visiblespectrum (e.g., LWIR) images. In relation to detection of airportlighting systems for landing site localization relative to airportlighting (as described further in relation to Blocks 220 and 220 bbelow), the camera subsystem can include and apply filtering (e.g.,through filtering optical elements, through operations in software,etc.) to received image data to detect spectra of light emitted fromairfield landing systems (e.g., lighting systems and/or markings inaccordance with Federal Aviation Administration Advisory Circular150/5345-46E, lighting systems and/or markings in accordance withInternational Civil Aviation Organization standards). As describedabove, in Blocks 210 a and 210 b, the computing system can receiveimages generated from a port side of the aircraft, a starboard side ofthe aircraft, a belly region of the aircraft, and/or a nose-region ofthe aircraft for landing site localization.

In relation to Blocks 210 a and 210 b, the computing system can receiveimages that have a characteristic resolution (e.g., associated with asensor size), aspect ratio, and/or directionality (e.g.,unidirectionality associated with 360 degree images), format colormodel, depth, and/or other aspects. The images can further include oneor more of: monoscopic images, stereoscopic images, panoramic images,and/or any other suitable type of image. Furthermore, while images aredescribed, the computing system associated with Blocks 210 a and 210 bcan receive video data and/or any other suitable type of data.

2.1.1 Method—Optional Submethod for Triggering Receipt of Image Data

FIG. 2C depicts a flowchart of a portion 260 of a method for landingsite localization, in accordance with one or more embodiments, where theportion 260 of the method functions to trigger at least one of imagecapture and image transmission upon detection of a level of proximitybetween the aircraft and the landing site, as shown in FIG. 2A. Theportion 260 of the method can reduce compute power (e.g., in relation todata processing and transmission, in relation to battery managementrequirements, etc.) that would otherwise be used to process images notcapturing the landing site within a field of view. In Block 261, thecomputing system receives a position output from a global positioningsystem (GPS) of the aircraft. The position output can be derived fromtransmissions between a satellite and a GPS onboard the aircraft anddescribe a geographic distance between the aircraft and the landingsite. In an example of Block 261, the computing system (e.g., navigationsubsystems of the system) can monitor the geographic location of theaircraft in near real time, calculate the distance between thegeographic location of the aircraft and the geographic location of thelanding site, and transmit the distance to the computing system of Block261 as the position output. In variations, the computing system canreceive 261 position outputs derived from other distance measuringapparatus, such as a transponder-based distance measuring equipment(DME), a non-directional beacon (NDB), a lateral navigation (LNAV)system, a vertical navigation (VNAV) system, or an area navigation(RNAV) system. Additionally or alternatively, in still other variations,a position output can be determined from dead reckoning using othersensors (e.g., IMU components, etc.), which can be beneficial inGPS-unavailable or GPS-denied scenarios. As such, the computing systemcan receive position outputs in terms of geographic distances,line-of-sight-distances, or in any other suitable format.

The landing site can be a paved runway (e.g., a runway in Class Bairspace, a runway in Class C airspace, a runway in Class D airspace, arunway in other airspace), a landing strip (e.g., paved, grass, dirt), awater landing site, a landing site on snow, a landing site on sand, orany other landing site associated with an approach pattern and/or glideslope. The landing site can alternatively be a landing site associatedwith vertical takeoff and landing (VTOL) operations, such as those usedby a helicopter or distributed electric propulsion (DEP) aircraft. Thelanding site can also have lighting systems and/or markings described inmore detail below in Section 2.2.

As shown in FIG. 2C, in Block 262, the computing system compares theposition output to a proximity condition characterizing proximity of theaircraft to the landing site. The proximity condition is a thresholdcondition describing how close the aircraft is to the landing site(e.g., in terms of geographic distance, in terms of line-of-sitedistance, etc.). As such, a distance extracted from the position outputis compared to a threshold distance. In examples, the proximitycondition can be associated with a threshold distance of 15 miles fromthe landing site, 10 miles from the landing site, 5 miles from thelanding site, or any other distance from the landing site. The thresholdcondition can additionally or alternatively be associated with entranceinto airspace associated with the landing site, arrival at a position(e.g., 45 degree entry position, crosswind position, downwind position,base position, final position, etc.) associated with an approach path tothe landing site. The threshold condition can additionally oralternatively be dynamically modified based on a speed of operation(e.g., cruise speed, approach speed, landing speed, etc.) of theaircraft, configuration of the aircraft (e.g., in terms of flapoperation, spoiler operation, landing gear operation, etc.) and/or aweather condition (e.g., associated with winds, visibility,precipitation, etc.). For instance, the threshold condition can be setto a greater distance threshold if the aircraft is moving at a fasterground speed.

Then, in Block 263, the computing system determines if the proximitycondition is met, and transitions a camera subsystem of the aircraft toan image capture mode. In determining satisfaction of the proximitycondition, the computing system can determine if the distance of theaircraft to the landing site, extracted from the position output, isless than, equal to, or greater than the threshold distance, and then aremote station or other portion of the computing system can generatecamera control instructions that are relayed to the camera subsystemthrough a camera control unit (e.g., portion of a flight computer)onboard the aircraft, in order to transition the camera subsystem to theimage capture mode (or an image transmission mode). Then, the remotestation of the computing system can receive the image from the camerasubsystem 264. Once the proximity condition is met, the camera subsystemcan be held in the image capture and/or transmission modes to transmitimages at a desired frequency to the remote station.

Additionally, an onboard computing system can instruct camera subsystem264 (or an image sensor system thereof) to capture an image using one ormore different image capture modes, in order to maximize visibility ofitems of interest in the image. For example, the system can take imagescorresponding to a range of exposure times, apertures, focal lengths,and/or any other suitable image-capture parameter. In relation tofiltering, if the camera subsystem 264 has selectable Bayer filters orother color filter arrays, color filtering can be applied during imagecapture as well. The selection criteria associated with different imagecapture modes can be chosen based upon one or more factors including:contrast information present in an image, time of day, sun ephemeris,moon ephemeris, weather conditions, airspeed, wind conditions, vibrationconditions, and/or any other suitable geospatial/atmospheric informationobservable or known by the onboard computing system.

2.2 Method—Determining Reference Features of Landing Site

As shown in FIGS. 2A and 2B, Blocks 220 a and 220 b includefunctionality for determining a reference position of a reference objectassociated with the landing site. In Blocks 220 a and 220 b, thecomputing system facilitates processing of the image(s) received fromthe camera subsystem of the aircraft at a location remote from theaircraft (e.g., at a remote station of the computing system), withmanual input by a human entity and/or automatically through imageprocessing and computer vision operations. In Blocks 220 a and 220 b,the remote station of the computing system can determine a singleposition or multiple reference positions of a single reference object ormultiple reference objects. As such, the relationships between thereference position(s) and the reference object(s) do not have to beone-to-one.

The reference object(s) associated with Blocks 220 a and 220 b caninclude stationary objects. Stationary objects can include approachlighting systems (e.g., visual approach slope indicator lights,precision approach path indicator lights, other approach lights), runwaylighting systems (e.g., lights associated with runway features, lightsassociated with clearances, lights associated with other air trafficcontrol instructions), taxiway lighting systems (e.g., lights associatedwith taxiway features, lights associated with clearances, lightsassociated with other air traffic control instructions, etc.), beacons,other airport lights, and/or other non-airport lights in the vicinity ofthe landing site. Airport lighting objects can be regulated objects(e.g., according to International Civil Aviation Organizationregulations).

The reference object(s) associated with Blocks 220 a and 220 b canadditionally or alternatively include airport markers associated withrunway markings (e.g., centerlines, hold short bars, runway numbers,displaced thresholds, etc.), taxiway markings (e.g., centerlines,approach to hold short bars, instrument landing system positionindicators, movement areas, non-movement areas, parking areas, etc.),airport signage, other airport markers, and/or other markers in thevicinity of the landing site. Airport markers can be regulated objects(e.g., according to International Civil Aviation Organizationregulations).

The reference object(s) associated with Blocks 220 a and 220 b canadditionally or alternatively include large scale landing site objects(e.g., runways, taxiways, buildings, fields, transportationinfrastructure, other infrastructure, geographic features, etc.), whereedges, corners, or any other suitable feature of the objects can bedetected and used as a landing site reference.

In determining the position(s) associated with the reference object(s),the remote station of the computing system can receive a packet from theentity, where the packet includes coordinates or other descriptors ofthe reference positions in space. As such, outputs of Block 220 a and220 b can characterize the position(s) of the reference object(s)associated with the landing site in a format that is computermachine-readable and able to be processed to produce additional outputsin downstream portions of the method.

Variations of manual, autonomous, and semi-autonomous aspects ofdetermining reference features of the landing site are described inSections 2.2.1-2.2.3 below.

2.2.1 Method—Determining Reference Features of Landing Site—OperatorInput

FIG. 3A depicts a schematic of a variation of 220 a and 220 b, where theimages(s) is/are transmitted to an entity at an operator interface, andupon selection of one or more positions of the reference objects(s)associated with the landing site captured in the images, the computingsystem receives packets characterizing the reference position(s) of thereference object(s). In more detail, for each image generated by thecamera subsystem, the computing system (e.g., remote station incommunication with a data center) transmits the image to a display ofthe operator interface for observation by the entity at the operatorinterface. Through input devices coupled to the display of the operatorinterface, the entity can then select positions and/or boundarylocations of the reference object(s), such as lighting at the landingsite, or boundaries of a runway (e.g., positions of corners of atrapezoid defining corners of the runway, positions of points along arunway centerline, etc.) at the landing site. As such, the computingsystem receives 320 a packets containing coordinates of the locations ofthe reference object(s) upon selection of the coordinates by the entitythrough the operator interface.

2.2.2 Method—Determining Reference Features of Landing Site—Autonomous

FIG. 3B depicts a schematic of another variation of 220 a and 220 b,where the images(s) is/are processed onboard automatically by the flightcomputer or transmitted to a remote computing entity that automaticallyapplies image processing operations to the image(s) and outputs packetscharacterizing one or more positions of the reference objects(s)associated with the landing site captured in the images. Data packetscan be stored in memory until used in downstream portions of the method.In relation to manual and semi-autonomous variations of referencefeature determination, data packets can be transmitted remotely to theremote station for observation by a manual operator (as in the fullymanual method) discussed previously. Regardless of where this method iscarried out, for each image generated by the camera subsystem, thecomputing system (e.g., remote station in communication with a datacenter, flight computer) applies a filtering operation to the image toextract or otherwise increase the intensity of features of the referenceobject(s) captured in the image. The filtering operation can be a colorfiltering operation that isolates image pixels associated with aparticular color (e.g., light wavelength, paint color, signage color,etc.) of the reference object to generate a color-filtered image. Thefiltering operation can also apply contrasting operations and/orsaturation increasing operations to increase the contrast prior to orpost application of a color filter. The filtering operation can alsostack or aggregate multiple images in another manner in order toincrease contrast.

After application of the filtering operation, the computing system thenapplies a centroid algorithm to identify a center position of thereference objects, which in the image shown in FIG. 3B include an arrayof light objects at the landing site. Alternative embodiments can omitapplication of a centroid algorithm and alternatively use anotherfeature extraction approach (e.g., speeded up robust feature approach,oriented FAST and rotated BRIEF approach, scale invariant featuretransform approach, etc.) locating a reference position of an objectwithin an image. The computing system can also apply a transformationoperation to transform the image (or filtered version thereof) from a 3Dspace to a 2D space, using a homography matrix operation, covariancematrix operation, or another transformation operation. The computingsystem can then automatically compare and match the centroid and/orextracted features of the transformed image to a database of airportlighting positions, including lighting positions at the landing site(e.g., using a random sample consensus operation, using an iterativeclosest point operation, etc.). In more detail, transformations of theimages can include scaling operations, perspective skewing operations,rotation operations, and/or any other suitable operations that transforminbound images to a form that maps onto a scale, perspective, rotation,or other format aspect of the images in the database of airport lightingpositions. Matching can then be performed between transformed imagesthat have analogous formats to the image formats in the database ofairport lighting positions. Outputs of the transformation and matchingprocesses are then used to generate data packets associated withcoordinates of the locations of the reference objects (e.g., lights). Assuch, the computing system receives 320 b packets containing coordinatesof the locations of the reference object(s) upon generation of thecoordinates in an automated manner using image processing and matchingoperations.

FIG. 3C depicts a schematic of an alternative variation to the variationof the portion of the method shown in FIG. 3B, where, in the alternativevariation the computing system and/or remote computing entity omitsapplication of a filtering operation to generate 320 c the packetincluding the position(s) of the reference object(s). In stillalternative variations, the computing system and/or remote computingentity can omit application of a transformation operation from 3D to 2Dspace in relation to comparing and matching images against a database ofairport reference objects.

2.2.3 Method—Determining Reference Features of LandingSite—Semi-Autonomous

FIG. 3D depicts a schematic of another variation of 220 a and 220 b,where the images(s) is/are transmitted to a remote computing entity thatautomatically applies image processing operations to the image(s), wherethe image processing operations can be similar to or identical to thoseperformed in relation to FIGS. 3B and 3C. After processing the images,the remote computing entity can then transmit digital content derivedfrom image processing to an entity at an operator interface. The entityat the operator interface can then verify the output of the remotecomputing entity, and upon verification by interaction with an inputdevice of the operator interface, the computing system can receivepackets characterizing one or more positions of the reference objects(s)associated with the landing site captured in the images.

2.3 Method—Generating Estimated Pose(s)

As shown in FIGS. 2A and 2B, Blocks 230 a and 230 b includefunctionality for generating an image-estimated pose of the aircraftfrom outputs of Blocks 220 a and 220 b, respectively. In Blocks 230 aand 230 b, the computing system (e.g., a portion of the computing systemoperating at the remote station, a portion of the computing systemoperating at a flight computer onboard the aircraft) receives thereference position(s) of the reference object(s) associated with thelanding site, and applies transformation processes to extract theimage-estimated pose of the aircraft at the time that the image wastaken. The reference position(s) can be transmitted to the computingsystem (or between portions of the computing system) with a uniqueidentifier matching the image, such that reference positions can beassociated with images and/or time stamps at which the images weretaken.

In one variation, in relation to outputs associated with FIG. 3A, thecomputing system can receive coordinates of bounding corners of therunway within the image, and apply a transformation operation from adimensional space associated with the image to a dimensional spaceassociated with the camera or sensor from which the image was taken(e.g., using a homography matrix operation, using a covariance matrixoperation, etc.). The transformation operation can include an imageregistration operation (e.g., registration with perspective projection,registration with orthogonal projection, etc.) to extract camera posefrom the reference positions of the bounding corners of the runway.Then, the computing system can perform a transformation operation fromthe dimensional space associated with the camera to the dimensionalspace associated with the pose of the aircraft to generate theimage-estimated pose of the aircraft. Outputs of Blocks 230 a and 230 bcan include, for each image processed, a position and orientation of theaircraft in 3D space, with global coordinates of the aircraft androtations about axes of the aircraft.

In another variation, in relation to outputs associated with FIGS. 3B,3C, and 3D, the computing system receives coordinates of a centroid ofan array of airport lights. The computing system can additionally oralternatively receive parameters (e.g., matrix element coefficients) ofa transformation process used in the comparing and matching operation ofBlocks 320 b, 320 c, and 320 d. Then, based on the centroid and/orparameters of the transformation process, the computing system can applya transformation operation from a dimensional space associated with thecentroid and/or database image to a dimensional space associated withthe camera or sensor from which the image was taken (e.g., using ahomography matrix operation, using a covariance matrix operation, etc.).The transformation operation can include an image registration operation(e.g., registration with perspective projection, registration withorthogonal projection, etc.) to extract camera pose from the referencepositions. Then, the computing system can perform a transformationoperation from the dimensional space associated with the camera to thedimensional space associated with the pose of the aircraft to generate330 b, 330 c, 330 d the image-estimated pose of the aircraft. Outputs ofBlocks 330 b-330 d can include, for each image processed, a position andorientation of the aircraft in 3D space, with global coordinates of theaircraft and rotations about axes of the aircraft.

2.3.1 Camera Calibration

In generating the image-estimated pose, the computing system canimplement a calibration operation on the camera subsystem in order togenerate image-estimated poses with a desired level of accuracy, and/orto inform decisions related to flight operations of the aircraft. Forinstance, an unsuccessful instance of the calibration operation can beused to transition the aircraft to a grounded status, thereby preventingflight of the aircraft until the reason for lack of success of thecalibration operation is determined and resolved.

FIG. 4A depicts an embodiment of a method 470 a for camera subsystemcalibration, in accordance with one or more embodiments, and FIG. 4Bdepicts a schematic of an embodiment of the method 470 b shown in FIG.4A. The methods shown in FIGS. 4A and 4B are used to precisely determinemounting positions and/or optical properties of camera subsystems used.

In Blocks 471 a and 471 b, the computing system (or other detectionsystem onboard the aircraft) detects one or more marker objects near theaircraft using near-field communication methods, optical detectionmethods, methods based on geographic position of the aircraft, or anyother suitable methods.

Then, in Blocks 472 a and 472 b, the camera subsystem captures andtransmits an image of the one or more marker objects near the aircraft(e.g., a path of motion of the aircraft, a parking position of theaircraft, etc.) and a computing system receives the image. The image isreceived and processed prior to takeoff of the aircraft (e.g., during apreflight inspection, during a taxiing operation of the aircraft), suchthat an informed flight decision can be made based on satisfactorycalibration of the camera subsystem. However, variations of the methods470 a and 470 b can additionally or alternatively perform instances ofthe calibration while the aircraft is in flight.

Given known features (e.g., positions, shapes, colors, heights relativeto the aircraft, heights relative to the camera subsystem, etc.) of theone or more marker objects, the computing system generates 473 a, 473 ban output analysis characterizing spatial configuration and operation ofthe camera subsystem upon processing a set of features of the markerobject(s) extracted from the calibration image against a set ofreference features of the marker object(s). Generating the outputanalysis can include generating 474 a, 474 b a camera position output, acamera orientation output, and a camera distortion output. In thisregard, the computing system can compare extracted features of themarker object(s) to one or more standard reference images of the markerobjects. For instance, standard reference images can be images taken ofthe marker objects from different heights and/or orientations. As such,generating the camera position output and the camera orientation outputcan include performing a transformation operation and/or matchingoperation from an image space associated with the calibration image to areference space associated with the standard reference image(s), inorder to extract the position and orientation of the camera used to takethe calibration image. Generating the camera distortion output caninclude applying algorithms to determine distortion or other artifacts(e.g., noise) in the image, due to hardware (e.g., mounting issues,vibration issues) and/or software issues.

Based on the output analysis, the computing system can then generate 475a, 475 b instructions for control of at least one of the camerasubsystem and the aircraft based upon the output analysis. For instance,control of the camera subsystem can include cleaning a sensor of thecamera, adjusting a mounting position of the camera (e.g., throughactuators and/or gimbals coupled to the camera), rebooting the camera,or re-performing the calibration operation due to an indeterminateresult. Control of the aircraft can include approving flight operationsand providing instructions (e.g., to an FMS) that allow the aircraft totransition to a flight operation mode, or prohibiting flight operations(e.g., grounding the aircraft) and providing instructions that preventthe aircraft from transitioning to a flight operation mode.

As such, generating the image-estimated pose(s) of the aircraft, asdescribed in Section 2.3 above, can include generating theimage-estimated pose(s) from the reference position(s) of the referenceobject(s) associated with the landing sight, and outputs of thecalibration operation, based on known spatial orientation and operationparameters of the camera subsystem determined from the calibrationoperation.

2.4 Method—Updating Pose Trajectory in Coordination with IMU Outputs

As shown in FIGS. 2A and 2B, once an image-estimated pose is generatedfor an image received from the camera subsystem, the computing systemthen updates 240 a, 240 b a pose trajectory (e.g., time series of poses)of the aircraft using measurements taken from one or more IMUs onboardthe aircraft. Blocks 240 a and 240 b thus function to, based on therecent image data from the aircraft taken during flight to the landingsite, generate and update a pose trajectory of the aircraft, bycombining image data of the landing site with IMU outputs. In moredetail, updating of pose estimates based on image-derived informationcan be used to mitigate drift in propagation of poses generated bydead-reckoning with IMU measurements. As introduced above, the IMUoutput can include a position value (e.g., associated with a globalreference frame), a velocity value, an orientation value, and a timestamp. IMU outputs can additionally include an altitude value, anangular rate value, and any other value of a parameter derived fromacceleration or angular rate data. As such, the image-estimated posescan be processed with IMU outputs to determine or update pose values ofthe aircraft relative to the landing site, as opposed to anotherreference.

The pose trajectory includes aircraft poses (e.g., coordinates inCartesian space with rotational coordinates about rotational axes)associated with time stamps of images taken by the camera subsystem. Thepose trajectory also includes aircraft poses associated with time stampsof IMU outputs (e.g., in relation to a buffer or data stream of IMUoutputs). The pose trajectory can also include aircraft posesinterpolated between time points of images and/or IMU outputs. The posetrajectory can also include poses extrapolated or projected to timepoints beyond time stamps associated with images and/or IMU outputs. Assuch, Blocks 240 a and 240 b can include functionality for adjusting orcorrect past poses associated with time points prior to a current stateof the aircraft, time points associated with a current state of theaircraft, and/or time points projected to times beyond a current stateof the aircraft.

As shown in FIG. 2B, the computing system can process an image-estimatedpose having a time stamp with an IMU output having the same or a similar(e.g., nearest) time stamp to compute an updated pose of the aircraftrelative to the a location of the landing site. Computing the updatedpose can include performing a vector, array, or matrix manipulationprocess (e.g., transformation, addition, etc.) to combine informationfrom an IMU output with information from the image-estimated pose. Inmore detail, if position or orientation components of theimage-estimated pose are more accurate than IMU measurements, theassociated image-estimated pose components can be used to replacecomponents of the matching IMU-based pose estimate. The computing systemcan then apply a forward propagation operation to IMU data (e.g., IMUdata having time stamps unassociated with images, IMU data having timestamps associated with images) until another image-estimated pose isprocessed, thereby updating the pose trajectory with informationcomputed from the image-estimated pose. Alternatively, image-estimatedpose components can be averaged with IMU measurements across a window oftime associated with the image-estimated pose, and forward propagationwith the averaging process can be performed until anotherimage-estimated pose is processed.

2.5 Method—Flight Control

As shown in FIGS. 2A and 2B, the method 200 can include functionalityfor controlling flight of the aircraft toward a flight path to thelanding site. Based on the image-estimated pose, an updated pose takinginto account image data and IMU data, and/or the updated posetrajectory, the computing system (e.g., a portion of the computingsystem operating at the remote station, a portion of the computingsystem operating at a flight computer onboard the aircraft, etc.)generates 250 a, 250 b instructions for flight control of the aircraftto the landing site. The flight computer or other computing componentscontrolling operation of flight control surfaces receive theinstructions and control operational configurations of one or morecontrol surfaces of the aircraft to maintain or redirect flight of theaircraft toward the landing site. As such, blocks 250 a and 250 binclude functionality for controlling flight of the aircraft toward theflight path upon transmitting the set of instructions to a flightcomputer of the aircraft and manipulating one or more flight controlsurfaces of the aircraft based on the set of instructions.

In Blocks 250 a and 250 b, the computing system (e.g., the flightcomputer) can use generated instructions to control configuration statesof one or more of: ailerons of the aircraft (e.g., to affect flightabout a roll axis), flaps of the aircraft (e.g., to affect rate ofdescent), elevators of the aircraft (e.g., to control flight about apitch axis), rudders of the aircraft (e.g., to control flight about ayaw axis), spoilers of the aircraft (e.g., to control lift of theaircraft), slats of the aircraft (e.g., to control angle of attack ofthe aircraft), air brakes (e.g., to control drag of the aircraft), trimsurfaces (e.g., to control trim of the aircraft relative to any axisand/or reduce system mechanical load), and any other suitable controlsurfaces of the aircraft.

In Blocks 250 a and 250 b, the computing system (e.g., the flightcomputer) can also use generated instructions to control configurationstates of power plant components including one or more of: manifoldpressure, revolutions (e.g., revolutions per minute), fuel mixture,electrical output from a battery, cooling system operational states(e.g., in relation to cowl flaps, in relation to liquid cooling systems,in relation to fins, etc.) for aircraft performance toward the landingsite.

In Blocks 250 a and 250 b, the computing system (e.g., the flightcomputer) can also use generated instructions to control other aircraftsystem aspects. For instance, the generated instructions can be used tocontrol communications with air traffic control at the landing site, inrelation to automated reception and/or read back of instructions fromair traffic control.

In relation to pose or pose trajectory of the aircraft, the computingsystem generates instructions that account for aircraft orientation dueto environmental effects and landing procedures due to environmentaleffects. For instance, the computing system can generate instructionsupon detecting crosswinds and computing a crosswind control factor forthe ailerons and rudders of the aircraft. In another example, computingsystem can generate instructions for a flight path to a preferred runwaydue to prevailing winds at the landing site (e.g., to avoid landing witha significant tail wind). In another example, the computing system cangenerate instructions for power plant settings in relation to winds atthe landing site.

In relation to pose or pose trajectory of the aircraft, the computingsystem can also generate instructions that account for landing sitefeatures and/or geographical features about the landing site. Forinstance, the computing system can generate instructions for producing asteeper or flatter approach (e.g., with slipped configuration settings,with flap settings, with landing gear settings, etc.) based on runwayfeatures (e.g., length, position relative to geography, positionrelative to obstacles along the approach path, etc.). In anotherexample, the computing system can generate instructions for controlsurface settings and/or power plant settings based on runway features,such as uphill grade, downhill grade, roughness, wetness, type (e.g.,grass, dirt, water, snow, etc.), width, and/or any other suitablelanding site feature. In another example, the computing system cangenerate instructions for control of the aircraft and/or verification ofappropriate pose relative to a desired runway, which can be beneficialif there are multiple parallel runways and/or taxiways about the desiredrunway for landing.

In relation to pose or pose trajectory of the aircraft, the computingsystem can also generate instructions that account for type of landinggear of the aircraft. For instance, the computing system can generateinstructions to maintain orientation for a three-point landing for anaircraft with a conventional landing gear configuration (e.g.,tailwheel). In another example, the computing system can generateinstructions to adjust orientation for a wheel landing for an aircraftwith a conventional landing gear configuration. In another example, thecomputing system can generate instructions to adjust orientation for anaircraft with a tricycle gear setting. In another example, the computingsystem can generate instructions to adjust orientation for an aircraftwith a crosswind landing gear.

However, the computing system can generate instructions used by theflight computer to control aircraft operation for other aircraftaspects, other environmental aspects, and/or other landing site aspects.

2.6 Method—Additional Blocks and Applications

The method can optionally include functionality for using animage-estimated pose, pose trajectory, or other method output to performa system check. FIG. 5 depicts a method for performing a system check,in accordance with one or more embodiments. As such, the computingsystem (e.g., portion at a remote station, portion onboard the aircraft)can generate 580 a system check output from a comparison between one ormore of the image-estimated pose and the pose trajectory, and an outputof another navigation system of the aircraft (e.g., a GPS), to evaluateperformance of the other navigation system. For instance, the computingsystem can compare a position component of a pose of the aircraftassociated with a given time stamp to a position of the aircraftdetermined from a GPS output at the given time stamp. The comparison canbe used to determine if the image-derived position is significantlydifferent from the GPS-derived position.

Then, based on the comparison, the computing system can generate 581instructions for control of the flight management system and/or flightcomputer of the aircraft, in relation to reliance upon the GPS or othernavigation, in relation to aircraft control, and/or for any othersuitable purpose. Aircraft control instructions can include variousinstructions described in Section 2.5 above, or any other suitableinstructions. Navigation system control instructions can includeinstructions for rebooting a navigation system, transitioning anavigation system to a deactivated or idle state, preventing anavigation system from controlling other aircraft subsystems (e.g., anautopilot system), and/or any other suitable navigation system controlinstructions.

Additionally or alternatively, the method and associated systemcomponents can include functionality for supporting a pilot operatingthe aircraft. For instance, the method and/or system can operate in aco-pilot operation mode where any generated analyses of pose, analysesof pose trajectory, and/or instructions are transformed intonotifications to the pilot (e.g., at a display, through an audio outputdevice, etc.) in relation to suggestions for control of the aircraft.Notifications can include a notification to abort landing (if landing isdeemed to be unsafe), a notification that indicates that the approach tothe landing site is appropriate, a notification related to changes tocourse of the aircraft (e.g., relative to the landing site), anotification related to configuration of the aircraft in relation toapproach to the landing site, and/or any other suitable notification.The method(s) described can, however, include any other suitable stepsor functionality for determining aircraft poses while the aircraft is inflight, controlling flight operation of the aircraft (e.g., toward alanding site), and/or evaluating performance of subsystems of theaircraft based on computed pose information.

3. Conclusion

The system and methods described can confer benefits and/ortechnological improvements, several of which are described herein. Forexample, the system and method employ non-traditional use of sensors(e.g., image sensors, IMUs, etc.) to determine poses of an aircraftwhile the aircraft is in flight toward a landing site. Landing anaircraft, in particular, requires dynamic monitoring and control ofaircraft operational states, and the method and system employ sensors ina novel manner for control of flight of aircraft (e.g., fixed wingaircraft, other aircraft) in relation to landing.

The system and method also reduces computing requirements and costsassociated with standard systems for guided landing. For instance, byusing images taken at a set of time points and IMU data, the systemachieves determination of aircraft pose and control of aircraft flightoperation with less data and computing power than other systems forautomated landing.

The system and method also include functionality for evaluatingperformance of other subsystems of the aircraft (e.g., image capturesubsystems, navigation systems, etc.) to improve their performance orotherwise improve safety of a flight operation.

The foregoing description of the embodiments has been presented for thepurpose of illustration; it is not intended to be exhaustive or to limitthe patent rights to the precise forms disclosed. Persons skilled in therelevant art can appreciate that many modifications and variations arepossible in light of the above disclosure.

Some portions of this description describe the embodiments in terms ofalgorithms and symbolic representations of operations on information.These algorithmic descriptions and representations are commonly used bythose skilled in the data processing arts to convey the substance oftheir work effectively to others skilled in the art. These operations,while described functionally, computationally, or logically, areunderstood to be implemented by computer programs or equivalentelectrical circuits, microcode, or the like. Furthermore, it has alsoproven convenient at times, to refer to these arrangements of operationsas modules, without loss of generality. The described operations andtheir associated modules may be embodied in software, firmware,hardware, or any combinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments may also relate to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, and/or it may comprise a general-purpose computingdevice selectively activated or reconfigured by a computer programstored in the computer. Such a computer program may be stored in anon-transitory, tangible computer readable storage medium, or any typeof media suitable for storing electronic instructions, which may becoupled to a computer system bus. Furthermore, any computing systemsreferred to in the specification may include a single processor or maybe architectures employing multiple processor designs for increasedcomputing capability.

Embodiments may also relate to a product that is produced by a computingprocess described herein. Such a product may comprise informationresulting from a computing process, where the information is stored on anon-transitory, tangible computer readable storage medium and mayinclude any embodiment of a computer program product or other datacombination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the patent rights. It istherefore intended that the scope of the patent rights be limited not bythis detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thepatent rights, one implementation of which is set forth in the followingclaims.

What is claimed is:
 1. A method for landing site localization, themethod comprising: in response to detecting an aircraft within aproximity threshold to a landing site, receiving an image taken from acamera subsystem coupled to the aircraft, the image capturing thelanding site within a field of view; transmitting the image to an entityremote from the aircraft; from the entity, receiving a packetcharacterizing a reference position of a reference object associatedwith the landing site; from the reference position, generating animage-estimated pose of the aircraft; updating a pose trajectory of theaircraft upon processing the image-estimated pose and an inertialmeasurement unit (IMU) output from an IMU coupled to the aircraft, theIMU output having a time stamp corresponding to the image; and basedupon the updated pose trajectory, generating an updated set ofinstructions for a flight computer of the aircraft, the set ofinstructions for flight control of the aircraft toward a flight path tothe landing site.
 2. The method of claim 1, wherein receiving the packetcomprises receiving a set of coordinate elements, selected by an inputdevice associated with the entity, the set of coordinate elementscorresponding to a boundary of the reference object.
 3. The method ofclaim 1, wherein generating the image-estimated pose comprises:generating a color-filtered image upon processing the image with a colorfilter corresponding to a color feature of the reference object;generating a transformed image upon transforming the color-filteredimage from a first dimensional space to a second dimensional space;extracting the reference position from the transformed image; andgenerating the image-estimated pose upon processing the referenceposition, the transformed image, and a set of images of candidatereference objects with a matching operation.
 4. The method of claim 3,wherein the reference object comprises a set of landing site lights. 5.The method of claim 1, wherein the reference object comprises at leastone of a landing site boundary, a pattern of landing site markers, and aset of landing site lights.
 6. The method of claim 1, wherein updatingthe pose trajectory comprises replacing components of the IMU outputwith components of the image-estimated pose.
 7. The method of claim 6,wherein updating the pose trajectory further comprises forwardpropagating components of the image-estimated pose to a set of IMUoutputs generated subsequent to the time stamp.
 8. The method of claim1, wherein receiving the image comprises receiving a longwave infrared(LWIR) image.
 9. The method of claim 1, wherein the camera subsystemcomprises: a first camera mounted to a port side of the aircraft and asecond camera mounted to a starboard side of the aircraft, wherein theimage comprises a pair of stereoscopic images, and wherein receiving thepacket characterizing the reference position comprises transmitting thepair of stereoscopic images to the entity through a user interfacedevice comprising a display, and receiving a selection of a boundingfeature corresponding to the reference position through the userinterface device.
 10. A method for landing site localization, the methodcomprising: receiving an image capturing a landing site from a field ofview of an aircraft; determining a reference position of a referenceobject associated with the landing site; from the reference position,generating an image-estimated pose of the aircraft; updating a posetrajectory of the aircraft upon processing the image-estimated pose andan inertial measurement unit (IMU) output from an IMU coupled to theaircraft, the IMU output having a time stamp corresponding to the image;and based upon the pose trajectory, generating a set ofcomputer-readable instructions for flight control of the aircraft towarda flight path to the landing site.
 11. The method of claim 10, whereingenerating the image-estimated pose comprises generating a transformedimage upon transforming the image from a first dimensional space to asecond dimensional space; and generating the image-estimated pose uponperforming a matching operation with the transformed image and a set ofimages of candidate reference objects.
 12. The method of claim 10,wherein the reference object comprises at least one of a landing siteboundary, a pattern of landing site markers, and a set of landing sitelights.
 13. The method of claim 10, wherein updating the pose trajectorycomprises processing components of the IMU output with components of theimage-estimated pose.
 14. The method of claim 10, wherein updating thepose trajectory comprises implementing a forward propagation operationwith the image-estimated pose and a second IMU output having a secondtime stamp different from the time stamp.
 15. The method of claim 10,further comprising controlling flight of the aircraft toward the flightpath upon transmitting the set of instructions to a flight computer ofthe aircraft and manipulating a flight control surface of the aircraftbased on the set of instructions.
 16. The method of claim 12, whereinreceiving the image comprises: receiving a position output from a globalpositioning system (GPS) of the aircraft; comparing the position outputto a proximity condition characterizing proximity of the aircraft to thelanding site; upon satisfaction of the proximity condition,transitioning a camera subsystem of the aircraft to an image capturemode; and receiving the image from the camera subsystem in the imagecapture mode.
 17. A system for landing site localization, the systemcomprising: a camera subsystem mounted to an aircraft with a field ofview of a flight path of the aircraft; an inertial measurement unit(IMU) mounted to the aircraft; a data transmission subsystem incommunication with the camera subsystem and the IMU; and a computingsystem comprising machine-readable instructions in non-transitory mediafor: transmitting, from the data transmission subsystem to a componentof the computing system remote from the aircraft, an image taken fromthe camera subsystem and capturing a landing site, receiving, at thecomponent, a reference position of a reference object associated withthe landing site within the image, from the reference position,generating an image-estimated pose of the aircraft, updating a posetrajectory of the aircraft upon processing the image-estimated pose andan output from the IMU having a time stamp corresponding to the image;and based upon the pose trajectory, generating a set of instructions forflight control of the aircraft toward a flight path to the landing site.18. The system of claim 17, wherein the camera subsystem comprises alongwave infrared (LWIR) sensor.
 19. The system of claim 17, wherein thecamera subsystem comprises a first camera mounted to a port side of theaircraft and a second camera mounted to a starboard side of theaircraft, wherein the image comprises a stereoscopic image, and whereinthe computing system comprises instructions for generating theimage-estimated pose of the aircraft from the stereoscopic image. 20.The system of claim 17, further comprising an electronic interfacebetween the computing system and a flight computer of the aircraft, theflight computer coupled to and controlling operational configurations ofcontrol surfaces of the aircraft, and the electronic interface operablein a mode that transmits the set of instructions to the flight computerand controls flight of the aircraft toward the flight path.